Designer quantum spin Hall phase transition in molecular graphene 
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Graphene was the first material predicted to be a time-reversal-invariant topological insulator; 
however, the insulating gap is immeasurably small owing to the weakness of spin-orbit interactions 
in graphene. A recent experiment [T] demonstrated that designer honeycomb lattices with graphene- 
like "Dirac" band structures can be engineered by depositing a regular array of carbon monoxide 
atoms on a metallic substrate. Here, we argue that by growing such designer lattices on metals or 
semiconductors with strong spin-orbit interactions, one can realize an analog of graphene with strong 
intrinsic spin-orbit coupling, and hence a highly controllable two-dimensional topological insulator. 
We estimate the range of substrate parameters for which the topological phase is achievable, and 
consider the experimental feasibility of some candidate substrates. 



The seminal work of Kane and Mele predicted the ex- 
istence of a new state of matter, the quantum spin Hall 
(QSH) insulator, in monolayer graphene sheets [2 j. The 
intrinsic spin-orbit interaction in graphene was shown to 
generate an energy gap in the nominally semi-metallic 
band structure, giving rise to an insulating state that 
is topologically distinct from a conventional band insu- 
lator. While this prediction spawned the field of time- 
reversal- invariant topological insulators, it was shown 
that graphene is a poor candidate material because the 
insulating bulk gap is extremely small, essentially due 
to the fact that the carbon atoms, being light elements, 
have a weak spin-orbit coupling strength [3j H] . A possi- 
ble way around this was found in Ref. [5 by depositing 
adatoms on the graphene surface though it is yet to be 
experimentally realized. Soon after the Kane-Mele pro- 
posal, HgTe/CdTe quantum wells were predicted [6] and 
experimentally confirmed [7] to exhibit the QSH insu- 
lating phase when the thickness of the HgTe layer was 
tuned properly. Although graphene has a small insu- 
lating gap, a realization of the the QSH insulator in 
graphene would have had certain advantages over the re- 
alization in HgTe/CdTe quantum wells, notably: (i) be- 
ing a purely two-dimensional system, graphene is accessi- 
ble to surface probes, unlike the quantum- well-based sys- 
tems; (ii) the phase transition from a trivial to a topo- 
logical phase in graphene can be tuned via an electric 
field during the course of an experiment, whereas in the 
quantum-well systems the tuning parameter is the HgTe 
layer-thickness, which cannot be tuned in-situ or even 
continuously. 

With the recent creation of "molecular graphene" a 
new window of opportunity exists to find the QSH insula- 
tor in a surface- accessible, electrically tuneable graphene 
analog pQ. Gomes et al. showed that when a triangu- 
lar lattice of repulsive carbon monoxide (CO) molecules 
is imposed on a two-dimensional electron gas (2DEG) 
(specifically the metallic surface states of copper), the 
electrons are confined to an artificial honeycomb lattice. 
In fact, the repulsive potential generated by a periodic 
arrangement of the CO molecules always acts to im- 
pose the dual lattice structure on the underlying elec- 



tron gas [8 . In particular, as mentioned, a triangu- 
lar array of molecules generates a honeycomb array of 
interstices, thus realizing the same lattice symmetries 
as those of graphene. Remarkably, the experiments of 
Ref. [1 showed, via scanning-tunneling microscopy, that 
the main features of the electronic structure of graphene 
(e.g., the massless Dirac dispersion) are also realized in 
the "molecular graphene" analog. 

In the present work we argue that the Kane-Mele QSH 
model can be realized in a molecular graphene system 
by depositing CO molecules on the surface of a suit- 
able heavy metal or semiconductor with more strongly 
spin-orbit coupled surface states than copper. We show 
that the microscopic spin-orbit coupling (SOC) on the 
surface of the heavy metal generates both the compet- 
ing "Rashba" and "intrinsic" spin-orbit coupling terms 
in the low-energy Hamiltonian of Kane and Mele. As the 
two low-energy terms depend differently on the micro- 
scopic SOC, their relative sizes (and therefore the nature 
of the ground state) can be tuned by altering the micro- 
scopic SOC via applied electric fields or tuning the CO- 
generated lattice structure. Thus, the system discussed 
in the present work realizes the field-tuned transition be- 
tween the topological insulator and the Dirac metal that 
was originally predicted by Kane and Mele [2 j. Further- 
more, as we discuss below, the exquisite tunability of 
molecular graphene makes it possible to study, for ex- 
ample, the physics of interfaces between topological and 
trivial insulators, in ways that have no parallel in con- 
ventional systems. 

We will consider a 2DEG formed by confined states 
on the (111) surface of a metal or semiconductor such 
as Ag, Au, or Bi. To be explicit let us consider the 
Au(lll) surface which exhibits surface states which are 
well-separated in energy from the bulk-states near the 
T-point of the Brillouin zone (BZ)[9 j. The surface states 
arise from sp-hybridized electronic orbitals and are well- 
described by a simple s and p-orbital tight-binding model 
for the surface orbitals from which, if we ignore SOC, a 
single, spin-degenerate band is extracted as a low-energy 
model[10 . At low energies this degenerate band is the 
only relevant surface state and can be accurately rep- 
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FIG. 1. Schematic illustration of proposed experimental 
setup, involving a triangular array of CO molecules (black 
circles) placed on a metallic substrate. The surface electrons 
of the metal are repelled by the CO molecules, and therefore 
occupy the sites of the dual lattice (red squares), which form 
a honeycomb lattice. 

resented in the isotropic, nearly- free-electron model [11] 
with the Hamiltonian 



where m* is an effective mass parameter. Following Refs. 
[TJ [12] , we consider the addition of a periodic potential 
induced by a triangular lattice of CO molecules placed 
on the (111) surface (see Fig. [I]). We assume that the 
electrons in the 2DEG can be taken to be noninteracting, 
so that the microscopic Hamiltonian we consider, T~L is of 
the form 

H = H + V CO (r) (2) 

where we approximate the molecular periodic potential 
Vco(r) by its lowest Fourier component, 

V CO (r) = y G ^.cos(G,-r), (3) 

where Vq > is repulsive and the two reciprocal 
lattice vectors for the triangular lattice are Gi = 
(f ) -^(1/2,-^/2) and G 2 = ) ^(1/2,^3/2) for 
CO molecules separated by a distance d. These reciprocal 
lattice vectors generate a hexagonal first BZ. 

The Hamiltonian 1-L has been carefully analyzed 
previously [TJ [12] [13] and, as we briefly recount now, gives 
rise to massless Dirac cones. In the nearly-free electron 
picture the potential Vco( r ) couples states with differ- 
ent momenta which are separated by reciprocal lattice 
vectors. To first order in Vg, one only needs to consider 
degenerate free-electron states that are coupled by a sin- 
gle reciprocal lattice vector G^. Thus, we focus on the 
neighborhood of the six ^-points at the corners of the 
first BZ (see Fig. [2^i,b). The six corners naturally decou- 
ple into two sets of three "equivalent" points connected 
by reciprocal-lattice vectors denoted A^A^ respectively 
for i = 1,2,3. The original eigenstates are modified by 
a non-vanishing Vg and the new eigenstates are formed 
from linear combinations of the Bloch waves at the three 



A^ or A{. We can thus perform degenerate state perturba- 
tion theory at these points with the effective Hamiltonian 

/e(Ai) V G V g \ 
H A =\ V G e(A 2 ) V G (4) 
V V G V g e(A 3 ) / 

where Ai,A 2 = Ai — Gi, A 3 = Ai + G 2 are one set of 
the BZ corners and e(A*) = ^ (|f) 2 . This Hamilto- 
nian has a Cs symmetry under permutation of the A^; 
the eigenstates break up into a singlet representation 
of Cs with energy E+ = e(A) + 2Vg and wavefunction 
\Kq) = -j= (1, 1, 1) , and a lower-energy doublet repre- 
sentation with energy E_ = e(A) — Vq and wavefunc- 
tions \K±) = ^= (l,e ±2i7r / 3 ,e ±4i7r / 3 ) . As shown in Refs. 
[TJ [12] and [13] if one projects the Hamiltonian onto the 
low-energy subspace spanned by \K±) and considers the 
k • P Hamiltonian perturbed around the A$ then one 
arrives at the massless Dirac Hamiltonian. If we com- 
bine the doublet \K±) from the A^ points with the dou- 
blet \K±) from the A^ points we arrive at the effective 
graphene-like Hamiltonian 

#Dirac = fiVf [k x G\ + ^T 3 Cr 2 ] (5) 

where vj = 3^^^, r 3 is a 2 x 2 Pauli matrix acting 
on the valley index (K,K f ), <j a are the Pauli matrices 
acting on the doublet ± index, and the tensor product is 
implicit. 

In addition to the effective projected model stemming 
from i^o, we wish to consider the effects of spin-orbit 
coupling on the low-energy electronic structure. With- 
out SOC (and without external magnetic fields) we find 
that i^Dirac is trivially extended to a spin-independent 
8x8 Hamiltonian which includes the spin- 1/2 degree 
of freedom. Electric fields generated by surface effects 
and the atomic crystalline potential will give rise to non- 
trivial spin-orbit effects in i^Dirac- Angle-resolved photo- 
emission experiments clearly show a large spin-splitting 
in the Au(lll) surface states due to a non- vanishing 
Rashba effect described by the free-electron Hamiltonian 
of the surface states with SOC 

iJ » soc) = ^ +afl(Pxs) ' 1 (6) 

where aR is the Rashba coefficient and s is a set of Pauli 
matrices representing physical spin- 1/2 [14]. Projecting 
this term onto the low-energy Dirac subspace involves 
certain subtleties, which arise because (as is well known) 
the effective SOC in a projected band arises from transi- 
tions to neighboring bands [15 j. Thus, since the symme- 
tries of Eqs. [5] and [6] are quite different, the form of the 
terms that can appear in Eq. [5] will potentially also be 
different. 

Thus, with these caveats in mind, we will derive the 
effective spin-orbit coupling seen by the low-energy Dirac 
Hamiltonian from "first principles" in the nearly-free 
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FIG. 2. Band structure of "molecular graphene" on a periodic (honeycomb- lattice) substrate, e.g., the surface of Au. (a) Recip- 
rocal lattice structure, showing the substrate reciprocal lattice (gray solid hexagons) as well as the first Brillouin zone (BZ) of 
the reciprocal "molecular graphene" superlattice (dashed shaded hexagons) near the F point — red dot — of each substrate unit 
cell. Solid and dashed arrows denote reciprocal lattice vectors of the substrate (Q») and of the superlattice (G») respectively. 
The two inequivalent Dirac cones are represented using triangles, (b) "Dirac" band structure for states at corners of the first 
superlattice BZ. The six corner momenta form two inequivalent groups (blue and red respectively A», A»); (c) states at the three 
momenta marked (e.g.) in blue hybridized via the lattice potential to form two low-lying states and one higher-energy state. 
All three states are coupled via the intraband SOC. (d) Schematic one-dimensional cut through the band structure, showing the 
band gaps due to the substrate lattice potential (~ W\) and superlattice potential (~ Vb), as well as the multiple superlattice 
BZs that fit within the first substrate BZ. We are primarily interested in states at the edges of the first superlattice BZ (shaded 
ellipse), which are connected by the interband spin-orbit coupling (SOC) to the corresponding momenta in the upper band. 



electron limit. From symmetry considerations we expect 
two types of spin-orbit terms will arise in Eq. [5] (i) 
a Rashba-type term arising from the inversion symmetry 
breaking on the Au substrate surface (ii) an intrinsic SOC 
term of the Kane-Mele type. As we will now show, in the 
nearly-free electron limit both of these terms appear nat- 
urally if we include the effects of the surface potential 



V suhs .(v) = Q(-z) 



Wn 



Wi V cos(Q i • r) 



(7) 



where 6(— z) is a step function (since we have cho- 
sen the Au to be in the region z < 0), and the peri- 
odic surface potential of the Au(lll) crystal substrate is 
taken to have a hexagonal surface BZ with the recipro- 
cal lattice vectors Q x = (^) ^= (1/2, -y/3/2) and Q 2 = 

JL(iy2, y/3/2) where a is the lattice constant of Au. 
The Wo term essentially represents the work-function due 
to charge accumulation near the Au(l 11) /vacuum inter- 
face and the W\ term is the periodic modulation induced 
by the crystalline potential of the Au atoms. A few notes 
are in order at this point: (i) instead of including the W\ 
term we could have reformulated our electronic struc- 
ture calculation using the sp-orbital tight-binding model 
of Ref. [10] which would naturally generate the higher 
surface-bands needed to properly account for the SOC, 
but we chose to stay with the nearly-free electron for- 
malism, where higher bands are generated by the Au 



lattice potential (ex Wi), for a unified presentation; (ii) 
we have assumed a hexagonal surface BZ geometry as in 
an Au(lll) crystal-surface 16] : this assumption simplifies 
our analysis but is not an essential feature, as it will not 
qualitatively affect our analysis if the BZ geometry is dif- 
ferent. We can see that the latter is true by noting that 
since a <C d then \Qi\ ^> \d\ so the effective Dirac struc- 
ture lies well within the first BZ of the Au(lll) periodic 
potential and thus the details of the substrate bandstruc- 
ture at the lattice scale cannot substantially affect i^Dirac 
(see Fig. [2ji). 

For free electrons the general microscopic expression 
for the SOC is 



Hsoc 



h 



subs. 



X P) • s 



(8) 



where ao is the bare spin-orbit coupling, P denotes the 
momentum, and s the physical spin. Using the form 
of Vsubs. m Eq. (|7|), we can split the contributions into 
nominally Rashba, and intrinsic pieces 

H R = ^W (z X P) • s 



"0, 



(9) 



Hj = -^(z x P) • s V. cos(Qi • r) + . . . (10) 
at, z — 

where £ is the localization length of the surface states 
along z, and the omitted terms in Hi involve in-plane 
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gradients of the surface periodic atomic potential which 
do not generate any nontrivial SOC terms within the 
nearly-free-electron scheme. Owing to the form of its 
spatial modulation, which depends on the large recipro- 
cal lattice vectors Q^(~ l/AK Hj only has no n- vanishing 
matrix elements between states in (what we call) differ- 
ent substrate bands. It is analogous to the microscopic 
SOC considered in the tight-binding model of graphene 
in Ref. [3] which takes into account transitions to other 
high-energy bands. 

To project these SOC terms onto the subspace of the 
Dirac Hamiltonian we will use perturbation theory in the 
strength of the SOC. To carry this out we must assume 
that the SOC energy scale is smaller than the energy 
scale that gives rise to the gapless Dirac structure set 
by Vg, and which in turn is assumed smaller than the 
free-electron energy scale set by h 2 \G\\ 2 /2m* . Thus, we 
treat the lattice as a perturbation acting on the free- 
electron states to generate the Dirac subspace, and then 
treat the SOC as a perturbation acting on the gapless 
Dirac Hamiltonian. The assumption that the SOC is 
weaker than the other energy scales is essential to our 
analysis; however, the assumption of a weak lattice can 
presumably be relaxed without changing our qualitative 
conclusions. 

We will proceed in two steps by projecting Hr first, 
and then Hi. Projecting Hr (which includes the dom- 
inant contributions from a Lowdin partitioning of the 
higher bands [15 ) we find contributions to an effective 
Rashba coupling and an effective Kane-Mele coupling in 
^Dirac of the form 



ttR 

H eff 



^rr (r 3 a 2 si + ais 2 ) 

Heff = ^IR r SO-3S3 



(11) 

(12) 



where X RR = (l/2)W o7 , A Jfl = and 7 = a \Ai\/£ 

where we have included the leading intra-substrate band 
contributions, dropped an unimportant constant in H^ff, 
and used the fact that the surface state wavefunctions 
exponentially decay into the bulk with a characteris- 
tic length £ (see Fig. for the origin of the intra- 
substrate band contributions). The terms generated by 
Hr within the low-energy theory are thus of two kinds: 
(i) a Rashba-term that is first-order in Wo and arises 
because Hr mixes the two low-lying states, and (ii) a 
Kane-Mele term that is second-order in Wo, and arises 
because of virtual transitions into higher bands (but still 
in the lowest substrate band) mediated by Hr. Note that 
when 7 Wo < 3Vg, which is the condition for validity of 
our perturbation treatment, Xir < Xrr; therefore, Hr 
on its own does not give rise to the QSH phase. Hav- 
ing addressed the terms generated by Hr, we now turn 
to the spatially modulated coupling Hi. As we have 
noted above, due to the spatial modulations in Hi only 
states between different substrate bands are connected 
(see Fig. ^p). Again, after projecting, we find contri- 
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FIG. 3. Phase diagram of "molecular graphene" as a function 
of the spin-orbit coupling (SOC) parameter 7, introduced in 
the main text, and the work function Wo of the substrate 
(which can be tuned to some extent via gating) . The quantum 
spin Hall insulator is achieved in the region shaded blue. 



butions to the Rashba and Kane-Mele terms with coeffi- 
cients Xri = 2WP7/A and Xn = 2W 1 2 7 2 /A respectively 
where A = h 2 \Q\ 2 /2m*. 

We can combine these two results to obtain the total ef- 
fective Rashba and Kane-Mele couplings Xr = Xrr-\-Xri 
and Xi = Xir + Xn. To observe a topological insulator 
phase we need Aj > X R (along with 7 < w^^m/a 
in order to stay within the effective low energy sector) 
which imposes the restriction 



l/2 + 2Wi/A 



< 7 < 



3V G 



-.(13) 



W /6V G + 2W 1 2 /W A " ' " Wo+4WiW /A* 

The inequality in Eq. [l3j which indicates that the dimen- 
sionless parameter 7 is the crucial control parameter, is 
the primary result of this work. There are three ways to 
optimize 7 = ao|Ai|/£ due to its parameter dependence: 

(i) use heavy atom materials with large atomic SOC ao; 

(ii) tune the lattice constant of the CO molecules to be 
shorter so that |Ai| increases; (hi) add surface dopants or 
electrostatic gating to optimize the effects of the surface 
potential in the z-direction on the surface state wavefunc- 
tions. For our choice of a step function potential this im- 
plies we should decrease £ so that more weight lies near 
the region of strong electric field (z = 0). In Fig. [3] we 
plot an approximate phase diagram as a function of Wo 
and 7 using the values A = 5eV, W\ = 2eV, Vq — O.leV. 
The phase diagram is shaded where Eq ( 13 ) holds and the 
decreasing (increasing) curve represents the right (left) 
side of the inequality. Note that, for these parameters, 
the work function Wo must be less than 0.4 eV in order 
for the QSH phase to be realizable; this value is much 
smaller than the work function of pure metals such as 
Au (Wo ~ 5 eV), but can presumably be tuned, espe- 
cially for a semiconductor substrate, either chemically 
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(see, e.g., Ref. [17 ) or via a strong perpendicular electric 
field. For an Au(lll) surface, and a CO molecule spacing 
of d « 3 A, we estimate 7 of order .02 which is probably 
too small to be in the QSH phase for realistic parameters. 
On the other hand, there are multiple other compounds 
(e.g. BiAg 2 [E], BiTel [19], or Bi x Pb!- x /Ag [20 J with 
similar surface band structure, and with a 7 > 0.2 for 
3A; these should be possible to tune to the QSH 
phase. One would prefer to have a system near the lower 
phase boundary so that the topological phase transition 
could be tuned using an electric gate to nominally adjust 
Wo or the surface-state parameter £ entering 7. 

We now turn to possible schemes for detecting and ma- 
nipulating the topological phase in molecular graphene. 
The detection method used in the experiments of Ref. pQ 
was scanning-tunneling microscopy (STM), which di- 
rectly probes the local density of states as a function of 
energy; STM can reveal many of the important features 
of the Kane-Mele model as well, such as the existence of a 
bulk gap and of topologically protected edge states. One 
can test whether the edge states are protected against 
backscattering, e.g., by placing "impurities" such as va- 
cancies on the edge (this is straightforward to do as the 
lattice is engineered in the first place) and using STM, 
either to detect whether the impurities gap out the edge 
states, or to measure the scattering matrix via the tech- 
niques in Ref. [21 . Additionally, the ease of engineering 
sharp interfaces between the topological insulator and a 
trivial insulating phase by increasing the CO spacing in 
a neighboring region will allow for a careful study of the 
topological interface states. 

To summarize, we have argued that molecular 
graphene grown on the surfaces of heavy metals or semi- 
conductors should realize the Kane-Mele model, owing to 
the strong spin-orbit coupling in the underlying metal. 
The topological phase of the Kane-Mele model can be 
realized in a parameter range that is experimentally vi- 
able and which can be optimized by tuning the lattice 
spacing, electrical gates, and atomic spin orbit coupling. 
The tunability and controllability offered by molecular 
graphene, relative to the existing (quantum- well-based) 
2D topological insulators, make it a particularly appeal- 
ing platform for studying various questions of practical 
or conceptual interest especially since it provides an open 
surface with which one can easily access the edge-state 
properties, and optimistically the ability to tune the QSH 



topological phase transition by gating. 
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